Udacity Self-Driving Car Engineer Nanodegree¶
Project: Traffic Sign Recognition Classifier¶
This project aims to build a classifier that is able to classify traffic sign with advanced computer vision technique and deep neural net work model.
Method¶
A overview of the method for the project can be found here.
Data set¶
The classifier is trained, validated and tested with over hundreds of thousands of examples among 43 classes of traffic signs from the data base of German Traffic Signs Recognition Benchmark GTSRB.
References¶
More technical details can be found at Sermanet and LeCun, 2011 Ciresan et al., 2012
Step 0: Load The Data¶
In [5]:
## Download and upzip data set, only need to be done once
## download
# curl -O https://d17h27t6h515a5.cloudfront.net/topher/2017/February/5898cd6f_traffic-signs-data/traffic-signs-data.zip
## unzip
# unzip traffic-signs-data.zip
In [6]:
## Import relavent libariries
import pickle
import numpy as np
import pandas as pd
import cv2
import tensorflow as tf
import time
import scipy.ndimage
import matplotlib.pyplot as plt
from tensorflow.contrib.layers import flatten
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
%matplotlib inline
In [7]:
## Load pickled data, read training, validating and testing data
training_file = 'train.p'
validation_file = 'valid.p'
testing_file = 'test.p'
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(validation_file, mode='rb') as f:
valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']
# Read signData
sign_file = 'signnames.csv'
with open(sign_file, mode = 'r') as f:
signData = pd.read_csv(f)
Step 1: Dataset Summary & Exploration¶
The pickled data is a dictionary with 4 key/value pairs:
'features'is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).'labels'is a 1D array containing the label/class id of the traffic sign. The filesignnames.csvcontains id -> name mappings for each id.'sizes'is a list containing tuples, (width, height) representing the original width and height the image.'coords'is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES
1. 1 Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas¶
In [8]:
## Examine basics of data set
# Number of training examples
n_train = X_train.shape[0]
# Number of validation examples
n_validation = X_valid.shape[0]
# Number of testing examples.
n_test = X_test.shape[0]
# Shape of an traffic sign image
image_shape = X_train.shape[1:3]
# Number unique classes/labels there are in the dataset.
n_classes = signData.shape[0]
# number for class, counts for training data'
trainClass, trainIdx, trainCounts = np.unique(y_train, return_index = True, return_counts = True)
validClass, validCounts = np.unique(y_valid, return_counts = True)
testClass, testCounts = np.unique(y_test, return_counts = True)
def plotSignExample(trainClass,trainCounts,trainIdx):
fig = plt.figure(figsize=(12,8))
for idxClass, count, idxStart in zip(trainClass,trainCounts,trainIdx):
idxRnd = np.random.randint(idxStart, idxStart + count, 1)
ax = fig.add_subplot(5, 10, idxClass+1)
plt.imshow(X_train[idxRnd].squeeze())
plt.show()
return fig
In [9]:
fig = plotSignExample(trainClass,trainCounts,trainIdx)
fig.savefig('examples/DataSetVisual.png')
print("Number of training examples =", n_train)
print("Number of validation examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes in sign system =", n_classes)
print("Number of classes in training data =", trainClass.shape[0])
print("Number of classes in validating data =", validClass.shape[0])
print("Number of classes in testing data =", testClass.shape[0])
print(signData['SignName'])
1. 2 Include an exploratory visualization of the dataset¶
Visualize the German Traffic Signs Dataset using the pickled file(s).
In [10]:
## Plot the distributions of classes in each data set
def plotDistOri(trainClass,validClass,testClass,trainCounts,validCounts,testCounts):
fig = plt.figure(figsize=(15,5))
barW = 0.3
plt.bar(trainClass - barW, trainCounts, width = barW, align = 'center')
plt.bar(validClass, validCounts, width = barW, align = 'center')
plt.bar(testClass + barW, testCounts, width = barW, align = 'center')
plt.xlabel('Class')
plt.ylabel('Number of examples')
plt.legend(['Train','Valid','Test'])
plt.show()
return fig
def plotDataOri(X,y,signData):
Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
nDemo = 10
for idxClass, count, idxStart in zip(Class,Counts,Idx):
print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
fig = plt.figure(figsize=(nDemo,2))
idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
for j in range(1, nDemo):
ax = fig.add_subplot(1, nDemo, j)
ax.imshow(X[idxRnd[j]])
plt.show()
In [11]:
## Plot individual examples in each class in training data
fig = plotDistOri(trainClass,validClass,testClass,trainCounts,validCounts,testCounts)
fig.savefig('examples/DataSetDistribution.png')
In [169]:
## Plot randomly selected examples in each class in training data
plotDataOri(X_train,y_train,signData)
Step 2: Design and Test a Model Architecture¶
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89.
To meet specifications, the validation set accuracy will need to be at least 0.93.
There are various aspects to consider when thinking about this problem:
- Neural network architecture (is the network over or underfitting?)
- Play around preprocessing techniques (normalization, rgb to grayscale, etc)
- Number of examples per label (some have more than others).
- Generate fake data.
References: LeCun.
2. 1 Pre-process the Data Set¶
Pre-processing techniques used: Histogram equalization, grayscale, normalizatin
In [12]:
## Define parameters for pre-processing
opt = {'gamma': 1.5, 'gray': True}
In [13]:
## Define pipeline of pre-process data, followed with helper functions
def preProcess(X, y, opt):
n = X.shape[0]
dim = np.asarray(X.shape)
gamma = opt['gamma']
if opt['gray'] == True:
dim[3] = 1
X_pre = np.zeros(dim,dtype = np.float32)
for i in range(0, n):
xTemp = X[i]
xTemp = gamma_correction(xTemp,gamma)
xTemp = hist_equalization(xTemp)
if opt['gray'] == True:
xTemp = rgb2gray(xTemp)
xTemp = scaleDown(xTemp)
if opt['gray'] == True:
X_pre[i] = xTemp[:,:,np.newaxis]
else:
X_pre[i] = xTemp
return X_pre, y
def scaleDown(img):
return np.float32(img/255)
# return np.float32((img - 128)/128)
def scaleUp(img):
return np.uint8(img*255)
# return np.uint8(img*128 + 128)
def gamma_correction(img,gamma):
img = img/255
img = cv2.pow(img,1/gamma)
return np.uint8(img*255)
def hist_equalization(img):
# hist equal in YUV space
imgYUV = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
# equalize the histogram of the Y channel
imgYUV[:,:,0] = cv2.equalizeHist(imgYUV[:,:,0])
# convert the YUV image back to RGB format
imgOut = cv2.cvtColor(imgYUV, cv2.COLOR_YUV2RGB)
return imgOut
def sharpen_img(img):
"""
Applies the Gaussian Blur on original image using openCV library
"""
gb = cv2.GaussianBlur(img, (3,3), 20)
return cv2.addWeighted(img, 2, gb, -1.5, 0)
def rgb2gray(img):
return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
def rgb2yuv(img):
return cv2.cvtColor(img,cv2.COLOR_RGB2YUV)
In [14]:
## Examine the pre-processed data
def plotDataPre(X, X_pre, y, signData, opt):
Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
nDemo = 10
for idxClass, count, idxStart in zip(Class,Counts,Idx):
print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
fig = plt.figure(figsize=(nDemo,2))
idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
for j in range(1, nDemo):
ax = fig.add_subplot(2, nDemo, j)
ax.title.set_text(str(idxRnd[j]))
ax.imshow(X[idxRnd[j]])
ax = fig.add_subplot(2, nDemo, j + nDemo)
if opt['gray'] == True:
ax.imshow(X_pre[idxRnd[j]].squeeze(), cmap = 'gray')
else:
ax.imshow(X_pre[idxRnd[j]])
plt.show()
def plotDemoPre(x,idx):
fig = plt.figure(figsize=(8,2));
plt.subplot(141); plt.imshow(x.squeeze());
plt.title('Original ' + str(idx))
xTemp = gamma_correction(x,2)
plt.subplot(142); plt.imshow(xTemp)
plt.title('Gamma correction')
xTemp = hist_equalization(xTemp)
plt.subplot(143); plt.imshow(xTemp)
plt.title('Histogram equal')
xTemp = rgb2gray(xTemp)
plt.subplot(144); plt.imshow(xTemp, cmap = 'gray')
plt.title('Grayscale')
plt.show()
return fig
In [16]:
## Pre-process traing, validating and testing data set
X_train_pre, y_train_pre = preProcess(X_train, y_train, opt)
X_valid_pre, y_valid_pre = preProcess(X_valid, y_valid, opt)
X_test_pre, y_test_pre = preProcess(X_test, y_test, opt)
X_train_pre.astype(np.float32);
X_valid_pre.astype(np.float32);
X_test_pre.astype(np.float32);
In [17]:
## Plot individual demo of pre-processing
# idxTest = [7774,7991,8109,8150]
idxTest = [8109,8150]
for i in range(len(idxTest)):
fig = plotDemoPre(X_train[idxTest[i]],idxTest[i])
title = 'examples/preProcessDemo' + str(idxTest[i]) +'.png'
fig.savefig(title)
In [170]:
## Plot all pre-processed training data
plotDataPre(X_train, X_train_pre, y_train, signData, opt)
In [171]:
## Plot all pre-processed validation data
plotDataPre(X_valid, X_valid_pre, y_valid, signData, opt)
2.2 Augment data¶
More data are adding by rotating the signs to balance the distribution of class of signs
In [126]:
## Define parameters
optAug = {'angMax':5, 'angMin':-5}
In [127]:
## Define data augmentation method, followed by helper functions for augmenting data
def AugmentData(X,y,optAug):
nClass, nIdx, nCounts = np.unique(y, return_index = True, return_counts = True)
compRatio = (max(nCounts) / nCounts - 1)/2
compRatio = compRatio.astype(np.int)
addCounts = nCounts * compRatio
# print(nCounts)
# print(addCounts)
dimX = np.asarray(X.shape)
dimY = np.asarray(y.shape)
for i in range(len(addCounts)):
idxS = nIdx[i]
idxE = nIdx[i] + nCounts[i]
if addCounts[i] != 0:
dimX[0] = addCounts[i]
dimY[0] = addCounts[i]
X_temp = np.zeros(dimX, dtype = np.float32)
y_temp = np.zeros(dimY, dtype = np.uint8)
idxAug = np.random.randint(idxS, idxE, size = addCounts[i])
thetaAug = np.random.randint(optAug['angMin'], optAug['angMax'], size = addCounts[i])
for j in range(idxAug.shape[0]):
X_temp[j,:,:,:] = augRotate(X[idxAug[j]], thetaAug[j])
y_temp[j] = y[idxAug[j]]
X_aug_temp = np.append(X[idxS:idxE], X_temp, axis = 0)
y_aug_temp = np.append(y[idxS:idxE], y_temp, axis = 0)
if i == 0:
X_aug = X_aug_temp
y_aug = y_aug_temp
else:
X_aug = np.append(X_aug, X_aug_temp, axis = 0)
y_aug = np.append(y_aug, y_aug_temp, axis = 0)
else:
if i == 0:
X_aug = X[idxS:idxE]
y_aug = y[idxS:idxE]
else:
X_aug = np.append(X_aug, X[idxS:idxE], axis = 0)
y_aug = np.append(y_aug, y[idxS:idxE], axis = 0)
print('Number of examples before augmentation: ' + str(X.shape[0]))
print('Number of examples after augmentation: ' + str(X_aug.shape[0]))
return X_aug, y_aug
def combineData(X1,X2,y1,y2):
nClass1, nIdx1, nCounts1 = np.unique(y1, return_index = True, return_counts = True)
nClass2, nIdx2, nCounts2 = np.unique(y2, return_index = True, return_counts = True)
dimX = np.asarray(X1.shape)
dimY = np.asarray(y1.shape)
# print(nIdx1)
# print(nIdx2)
# print(nCounts1)
# print(nCounts2)
dimX[0] = X1.shape[0] + X2.shape[0]
dimY[0] = y1.shape[0] + y2.shape[0]
X = np.zeros(dimX, dtype = np.float32)
y = np.zeros(dimY, dtype = np.uint8)
# print(X1.shape)
# print(X2.shape)
# print(X.shape)
idx = 0
for i in range(len(nClass1)):
# print(str(i) + 'th segment starts at' + str(idx))
idxS1 = nIdx1[i]
idxE1 = nIdx1[i] + nCounts1[i]
idxS2 = nIdx2[i]
idxE2 = nIdx2[i] + nCounts2[i]
X[idx:idx+nCounts1[i]] = X1[idxS1:idxE1]
y[idx:idx+nCounts1[i]] = y1[idxS1:idxE1]
# print(str(y[idx]) + ' and ' + str(y[idx+nCounts1[i]-1]))
idx = idx + nCounts1[i]
X[idx:idx+nCounts2[i]] = X2[idxS2:idxE2]
y[idx:idx+nCounts2[i]] = y2[idxS2:idxE2]
# print(str(y[idx]) + ' and ' + str(y[idx+nCounts2[i]-1]))
idx = idx + nCounts2[i]
# print(str(i) + 'th segment finishes at' + str(idx - 1))
return X, y
def splitData(X,y):
X_train, X_valid, y_train, y_valid = train_test_split(X,y,test_size=0.2,random_state=888)
return X_train, X_valid, y_train, y_valid
def augRotate(x, theta):
xNew = scipy.ndimage.rotate(x, theta, reshape=False)
return xNew
def augShift(x, shift):
xNew = scipy.ndimage.shift(x, shift)
return xNew
In [149]:
## Define plot function
def plotDistAug(y, y_aug):
augClass, augIdx, augCounts = np.unique(y_aug, return_index = True, return_counts = True)
oriClass, oriIdx, oriCounts = np.unique(y, return_index = True, return_counts = True)
fig = plt.figure(figsize=(15,5))
barW = 0.4
plt.bar(oriClass - barW, oriCounts, width = barW, align = 'center')
plt.bar(augClass, augCounts, width = barW, align = 'center')
plt.xlabel('Class')
plt.ylabel('Number of examples')
plt.legend(['Original data set','Augmented data set'])
plt.show()
return fig
def plotDataAug(X,y,signData):
Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
nDemo = 10
for idxClass, count, idxStart in zip(Class,Counts,Idx):
print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
fig = plt.figure(figsize=(nDemo,2))
idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
for j in range(1, nDemo):
ax = fig.add_subplot(1, nDemo, j)
ax.imshow(X[idxRnd[j]].squeeze(), cmap = 'gray')
plt.show()
def plotDemoAug(x,idx):
x = scaleUp(x)
xAug = augRotate(x[0],10)
fig = plt.figure(figsize=(4,2));
plt.subplot(121); plt.imshow(x.squeeze(),cmap = 'gray');
plt.title('Pre-processed ' + str(idx))
plt.subplot(122); plt.imshow(xAug.squeeze(),cmap = 'gray');
plt.title('Augmented' + str(idx))
return fig
In [145]:
## Generating augmented data
print('Combine original training and validation data set')
X_total_pre, y_total_pre = combineData(X_train_pre, X_valid_pre, y_train_pre, y_valid_pre)
print('Generating augmented data')
X_total_aug, y_total_aug = AugmentData(X_total_pre, y_total_pre, optAug)
print('Spliting augmented data into training and validation set')
X_train_aug,X_valid_aug,y_train_aug,y_valid_aug = splitData(X_total_aug,y_total_aug)
print('Dimension of augmented training data is ' + str(X_train_aug.shape))
print('Dimension of augmented validation data is ' + str(X_train_aug.shape))
# print('For validation set')
# X_valid_aug, y_valid_aug = AugmentData(X_valid_pre, y_valid_pre, optAug)
# print('For testing set')
# X_test_aug, y_test_aug = AugmentData(X_test_pre, y_test_pre, optAug)
In [146]:
## Plot demo of augmented data
idxAugDemo = [7991]
for i in range(len(idxAugDemo)):
fig = plotDemoAug(X_train_pre[idxAugDemo],idxAugDemo)
title = 'examples/AugDemo' + str(idxAugDemo[i]) +'.png'
plt.savefig(title)
In [150]:
## Plot augmented data distributions
fig = plotDistAug(y_total_pre, y_total_aug)
fig.savefig('examples/AugDataDist.png')
In [138]:
## Plot combined pre-processed training and validation data
plotDataAug(X_total_pre, y_total_pre, signData)
In [139]:
## Plot augmented pre-processed total data set
plotDataAug(X_total_aug, y_total_aug, signData)
In [ ]:
## Plot augmented training data
plotDataAug(X_train_aug, y_train_aug, signData)
In [ ]:
## Plot augmented validation data
plotDataAug(X_valid_aug, y_valid_aug, signData)
2. 2 Model Architecture¶
In [151]:
## Define model parameters
optMod = {'epochs': 50, 'batchSize': 128, 'learnRate':0.0001,
'keepRate':0.8, 'dataChoice': 'aug',
'initMu': 0, 'initStd': 0.1}
In [152]:
## Define each individual layer
def convLayer(x, fSize, stride, inDepth, outDepth, optMod):
conW = tf.Variable(tf.truncated_normal(shape = (fSize, fSize, inDepth, outDepth),
mean = optMod['initMu'], stddev = optMod['initStd']))
conB = tf.Variable(tf.zeros(outDepth))
con = tf.nn.conv2d(x, conW, strides = [stride, stride, stride, stride],
padding = 'VALID') + conB
con = tf.nn.relu(con)
return con
def maxPool(x, fSize):
return tf.nn.max_pool(x, ksize = [1, fSize, fSize, 1], strides = [1, fSize, fSize, 1],
padding = 'VALID')
def fullCon(x, inDim, outDim, optMod):
W = tf.Variable(tf.truncated_normal(shape = (inDim, outDim), mean = optMod['initMu'],
stddev = optMod['initStd']))
B = tf.Variable(tf.zeros(outDim))
y = tf.matmul(x, W) + B
y = tf.nn.relu(y)
return y
In [153]:
## Construct full model
def fullModel(x, optMod):
# input dimension = [32, 32, 1]
convL1 = convLayer(x, 5, 1, 1, 70, optMod)
# output dimension = [28, 28, 70]
maxPool1 = maxPool(convL1, 2)
# output dimension = [14, 14, 70]
convL2 = convLayer(maxPool1, 5, 1, 70, 140, optMod)
# output dimension = [10, 10, 140]
maxPool2 = maxPool(convL2, 2)
# output dimension = [5, 5, 140]
convL3 = convLayer(maxPool2, 3, 1, 140, 210, optMod)
# output dimension = [3, 3, 210]
# maxPool3 = maxPool(convL3, 2)
# output dimension = [, , 210]
fCon1 = flatten(convL3)
fCon1 = tf.nn.dropout(fCon1, optMod['keepRate'])
# output dimension = 3*3*210
fCon2 = fullCon(fCon1, 3*3*210, 300, optMod)
# output dimension = 300
logits = fullCon(fCon2, 300, 43, optMod)
# output dimension = 43
return logits
2.4 Train, Validate and Test the Model¶
A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.
In [155]:
## Define training & validation data
if optMod['dataChoice'] == 'aug':
X_train_mod = X_train_aug
y_train_mod = y_train_aug
X_valid_mod = X_valid_aug
y_valid_mod = y_valid_aug
print('The augmented training and validation data are used')
elif optMod['dataChoice'] == 'pre':
X_train_mod = X_train_pre
y_train_mod = y_train_pre
X_valid_mod = X_valid_pre
y_valid_mod = y_valid_pre
print('The original training and validation data are used')
elif optMod['dataChoice'] == 'hyb':
X_train_mod = X_train_aug
y_train_mod = y_train_aug
X_valid_mod = X_valid_pre
y_valid_mod = y_valid_pre
print('The hybrid training and validation data are used')
In [156]:
## Set up tensor flow training session
# Set up place holder for input data
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.uint8, (None))
one_hot_y = tf.one_hot(y, n_classes)
keep_prob = tf.placeholder(tf.float32)
logits = fullModel(x, optMod)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = optMod['learnRate'])
training_operation = optimizer.minimize(loss_operation)
In [157]:
## Set up tensor flow session to evaluate accuracy and loss
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()
def evaluate(X_data, y_data, keepRate):
num_examples = X_data.shape[0]
total_accuracy = 0
total_loss = 0
# sess = tf.get_default_session()
for offset in range(0, num_examples, optMod['batchSize']):
batch_x, batch_y = X_data[offset:offset+optMod['batchSize']], y_data[offset:offset+optMod['batchSize']]
[accuracy, loss] = sess.run([accuracy_operation, loss_operation],
feed_dict={x: batch_x, y: batch_y, keep_prob: keepRate})
total_accuracy += (accuracy * len(batch_x))
total_loss += (loss * len(batch_x))
return (total_accuracy / num_examples, total_loss / num_examples)
In [158]:
# Plots history of learning curves for a specific model.
def plot_learning_curves(train_loss, valid_loss, train_acc, valid_acc):
"""
Plots learning curves (loss and accuracy on both training and validation sets).
"""
epoch = range(1, int(train_loss.shape[0])+1)
fig = plt.figure(figsize = (10, 4))
axis = fig.add_subplot(1, 2, 1)
axis.plot(epoch,train_acc*100)
axis.plot(epoch,valid_acc*100)
axis.legend(['train','valid'])
plt.grid()
plt.xlabel("epoch")
plt.ylabel("accuracy")
plt.ylim(50, 110)
axis = fig.add_subplot(1, 2, 2)
axis.plot(epoch,train_loss)
axis.plot(epoch,valid_loss)
axis.legend(['train','valid'])
plt.grid()
plt.xlabel("epoch")
plt.ylabel("loss")
plt.ylim(0.001, 1)
return fig
In [159]:
## Train model and compute training and cross-validation error
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
num_examples = X_train_mod.shape[0]
train_loss_history = np.empty([0], dtype = np.float32)
train_accuracy_history = np.empty([0], dtype = np.float32)
valid_loss_history = np.empty([0], dtype = np.float32)
valid_accuracy_history = np.empty([0], dtype = np.float32)
print("Training...")
print()
for i in range(optMod['epochs']):
timeStart = time.time()
X_train_rand, y_train_rand = shuffle(X_train_mod, y_train_mod)
for offset in range(0, num_examples, optMod['batchSize']):
end = offset + optMod['batchSize']
batch_x, batch_y = X_train_rand[offset:end], y_train_rand[offset:end]
sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: optMod['keepRate']})
[train_accuracy, train_loss] = evaluate(X_train_mod, y_train_mod, 1)
[valid_accuracy, valid_loss] = evaluate(X_valid_mod, y_valid_mod, 1)
valid_loss_history = np.append(valid_loss_history, [valid_loss])
valid_accuracy_history = np.append(valid_accuracy_history, [valid_accuracy])
train_loss_history = np.append(train_loss_history, [train_loss])
train_accuracy_history = np.append(train_accuracy_history, [train_accuracy])
timeElapsed = time.time() - timeStart
print("EPOCH %.0f, time used = %.1f sec" % (i+1, timeElapsed))
print("Training accuracy = %.1f%%, training loss = %.3f" % (100*train_accuracy, train_loss))
print("Validation accuracy = %.1f%%, validation loss = %.3f" % (100*valid_accuracy, valid_loss))
print()
saver.save(sess, './TrafficSignModelV7')
print("Model saved")
In [160]:
## Monitoring training and validation accuracy
fig = plot_learning_curves(train_loss_history, valid_loss_history, train_accuracy_history, valid_accuracy_history)
fig.savefig('examples/TrainingHistory.png')
In [161]:
with tf.Session() as sess:
# new_saver = tf.train.import_meta_graph('TrafficSignModel.meta')
# new_saver.restore(sess, tf.train.latest_checkpoint('.'))
saver.restore(sess,tf.train.latest_checkpoint('.'))
[test_accuracy, test_loss] = evaluate(X_test_pre, y_test_pre, 1.0)
print("Testing accuracy = %.1f%%, testing loss = %.3f" % (100*test_accuracy, test_loss))
Step 3: Test a Model on New Images¶
To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.
You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.
Load and Output the Images¶
In [162]:
## Load the images and plot them here.
import os
import PIL
from PIL import Image
In [163]:
def fetchNewData(lists):
print(lists)
i = 0
X = np.empty([0, 32, 32, 3], dtype = np.uint8)
for list in lists:
image = Image.open('newTest/' + list)
image = image.resize((32,32))
xTemp = np.asarray(image, dtype=np.uint8)
xTemp = xTemp[np.newaxis]
X = np.append(X, xTemp, axis = 0)
i = i + 1
return X
def plotDataTest(X_new,y_new):
i = 0
fig = plt.figure(figsize=(20,3))
for list in lists:
image = Image.open('newTest/' + list)
plt.subplot(1,10,i+1)
image = image.resize((32,32))
plt.imshow(image)
xTemp = np.asarray(image, dtype=np.uint8)
plt.title('Class ' + str(y_new[i]))
xTemp = xTemp[np.newaxis]
X_new = np.append(X_new, xTemp, axis = 0)
i = i + 1
return fig
In [164]:
lists = os.listdir("newTest/")
# y_new = np.asarray([16,1,33,11,38,18,12,25,35], dtype=np.uint8)
y_new = np.asarray([12,1,25,18,11,33,35,38,16], dtype=np.uint8)
X_new = fetchNewData(lists)
fig = plotDataTest(X_new,y_new)
fig.savefig('examples/NewTestData.png')
Predict the Sign Type for New Images and Analyze Performance¶
In [165]:
## Run the predictions here and use the model to output the prediction for each image.
X_new_pre, y_new_pre = preProcess(X_new, y_new, opt)
In [166]:
with tf.Session() as sess:
saver.restore(sess, tf.train.latest_checkpoint('.'))
[testNew_accuracy, testNew_loss] = evaluate(X_new_pre, y_new_pre, 1.0)
print("New festing accuracy = %.1f%%, testing loss = %.3f" % (100*testNew_accuracy, testNew_loss))
Output Top 5 Softmax Probabilities For Each Image Found on the Web¶
For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.
In [167]:
## Define helper function to print out the top ten softmax probabilities for the new test images
def plotDemoTopK(X, y, probTopK, idxTopK, nCase):
fig = plt.figure(figsize=(13,15))
fig.subplots_adjust(hspace= 0, wspace = 1)
for i in range(nCase):
ax = fig.add_subplot(3,3,i+1)
ax.imshow(X[i])
label = topKLabel(y[i], probTopK[i],idxTopK[i])
plt.xlabel(label,fontsize = 12)
plt.show
return fig
def topKPredict(X, k_top):
sess = tf.get_default_session()
probMatrix = sess.run(tf.nn.softmax(logits), feed_dict = {x: X, keep_prob: 1.0})
topK = tf.nn.top_k(probMatrix, k = k_top)
return sess.run(topK)
def topKLabel(idxCorr,prob, idx):
s1 = str('Correct label is ' + signData['SignName'][idxCorr] + '\n')
for j in range(len(prob)):
s2 = (str(round(prob[j]*100,4)) + ' % being ' + signData['SignName'][idx[j]] + '\n')
s1 = s1+s2
return s1
In [168]:
nTopClass = 5
with tf.Session() as sess:
saver.restore(sess, tf.train.latest_checkpoint('.'))
topK = topKPredict(X_new_pre, nTopClass)
fig = plotDemoTopK(X_new,y_new,topK.values, topK.indices, len(y_new))
fig.savefig('examples/TopKDemo.png')
Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.
Step 4 (Optional): Visualize the Neural Network's State with Test Images¶
This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.
Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.
For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End Deep Learning for Self-Driving Cars in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.
In [ ]:
### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.
# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry
def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
# Here make sure to preprocess your image_input in a way your network expects
# with size, normalization, ect if needed
# image_input =
# Note: x should be the same name as your network's tensorflow data placeholder variable
# If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
featuremaps = activation.shape[3]
plt.figure(plt_num, figsize=(15,15))
for featuremap in range(featuremaps):
plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
if activation_min != -1 & activation_max != -1:
plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
elif activation_max != -1:
plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
elif activation_min !=-1:
plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
else:
plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")